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Information  is  incomplete  and  often  contradictory  re 
garding  the  extent  to  which  various  categories  of  raters 
agree  on  ratings  of  counselor  effectiveness.     The  primary 
purpose  of  the  present  study  was  to  examine  the  relation- 
ships among  University  of  Florida  student  counselors' 
self-ratings,  peer  ratings,  supervisor  ratings  and  client 
ratings  on  the  Counselor  Effectiveness  Scale   (CES) ,  Form 
2.     This  study  was  undertaken  to  clarify  the  use  of 
ratings  in  counselor  effectiveness  research. 

Twenty-seven  supervisors  in  the  Department  of  Coun- 
selor Education  at  the  University  of  Florida  and  in 
various  counselor  education  practicum  and  internship 
agencies  each  rated  a  randomly  selected  counselor  educa- 
tion graduate  student  supervisee  on  the  Counselor 
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Effectiveness  Scale,  Form  2.     These  27  student  counselor 
supervisees  also  were  rated  by  themselves,  27  randomly 
selected  peer  student  counselors,  and  by  every  client 
counseled  by  a  supervisee  for  a  predetermined  week.  The 
relationships  among  groups  for  both  overall  and  specific 
item  CES  ratings  were  ascertained  by  use  of  Pearson 
product-moment  correlations.     A  one-way  analysis  of 
variance  was  employed  to  test  whether  the  mean  ratings 
for  each  rating  group  differed  significantly  from  those 
of  the  other  rating  groups. 

The  major  finding  in  the  present  study  was  that 
there  were  no  significant  relationships  among  the  over- 
all counselor  effectiveness  ratings  made  by  counselors 
themselves,  peers,  supervisors,  and  clients.     In  addi- 
tion, no  difference  was  found  among  the  four  group  mean 
ratings  on  the  total  score  of  the  Counselor  Effectiveness 
Scale.     In  other  words,  no  significant  relationships  were 
found  between  pairs  of  rating  groups  in  terms  of  rating 
agreement,  nor  were  the  mean  overall  student  counselor 
ratings  made  by  each  group  significantly  different. 

Of  150  pairwise  correlations  for  the  25  item  scores, 
"nine  were  found  to  be  statistically  significant  (one 
px  .01,  eight  p  <  .05).     However,  because  of  the  great 
likelihood  of  alpha  error,  no  attempt  was  made  to  draw 
inferences  from  any  correlations  for  specific  items. 


viii 


CHAPTER  I 


PROBLEM  AND  PURPOSE  OF  THE  STUDY 
Purpose 

The  present  study  investigated  the  extent  to  which 
four  different  groups  of  raters  agreed  on  ratings  of 
counselor  effectiveness.     More  specifically,  the  primary 
purpose  of  this  study  was  to  investigate  the  relation- 
ships among  counseling  effectiveness  ratings  made  by 
peers,  supervisors,  clients,  and  student  counselors  them- 
selves on  a  counselor  effectiveness  rating  scale.  The 
secondary  purpose  of  the  study  was  to  examine  the  con- 
sistency of  ratings  made  by  these  groups  of  raters  on  the 
specific  items  within  the  scale. 

Background 

Counselor  effectiveness  research  has  important  im- 
plications for  counselor  selection   (ACES,  1967),  evalua- 
tion (Shoben,  1953;  Stablein,  1962),  and  prediction  of 
counseling  ability  (Walton  and  Sweeney,  1969) .  However, 
methods  of  determining  and  predicting  counselor  effec- 
tiveness remain  open  to  question. 

Intellectual  determinants  of  effective  counseling 
have  proved  less  than  satisfactory.     Previous  studies 
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have  indicated  no  relationship  (Carkhuff,  Piaget,  and 
Pierce,  1967)  or  a  negative  relationship   (Bergin  and 
Soloman,  1963)  between  rated  facilitative  functioning 
and  grade  point  average.     Furthermore,  the  Graduate  Record 
Examination  appears  to  be  of  dubious  value  in  predicting 
counselor  effectiveness  (Wittmer  and  Lister,  1971) . 

In  1963  the  American  Personnel  and  Guidance  Associa- 
tion recommended  the  examination  of  personal  as  well  as 
intellectual  qualities  of  counselor  candidates  (APGA, 
1963)  .     About  that  time  a  great  deal  of  research  was  done 
on  personality  correlates  of  counselor  effectiveness 
(Brams,  1961;  Kemp,  1962;  Stefflre,  King,  and  Leafgren, 
1962;  Combs  and  Soper,  1963;  Dole,  1964;  Russo,  Kelz, 
and  Hudson,  1964;  Wasson,  1965;  Wicas  and  Mahan,  1966). 

Generally,  the  criteria  of  effective  counseling  in 
such  research  were  ratings.     A  review  of  the  literature 
also  reveals  that  various  types  of  ratings  were  used  in 
most  studies  evaluating  the  effectiveness  of  counselors. 
Similarly,  a  great  many  tests  which  purported  to  predict 
counselor  effectiveness  were  validated  by  the  extent  to 
which  they  agreed  with  various  types  of  counselor  effec- 
tiveness ratings   (Walton  and  Sweeney,  1969) . 

Statement  of  the  Problem 

While  ratings  are  the  most  frequently  used  of  all  the 
criterion  measures  in  counselor  effectiveness  research, 
the  identity  of  those  doing  the  rating  varies  widely. 
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Groups  rating  counselor  effectiveness  in  previous  research 
include  supervisors,  counselors,  peers,  clients,  coached 
clients,  and  administrators.     Many  researchers  contend 
that  each  of  these  groups  can  provide  valid  ratings  of 
counselor  effectiveness.     However,  others  suggest  that 
different  groups  of  people  see  counseling  differently, 
and  therefore  may  rate  counselor  effectiveness  different- 
ly.    Factors  hypothesized  to  relate  to  differential 
ratings  include  differing  definitions  of  counseling 
(Knupfer,  Jackson,  and  Krieger,  1959),  priorities  (John- 
ston, 1966),  outside  cues   (Kiesler,  1966),  internal 
references  or  anchors  (Klein  and  Cleary,  1967) ,  biases 
(Patterson,  1967),  dependent  judgments   (Chinsky  and 
Rappaport,  1970) ,  and  theoretical  orientations  (Ward, 
1974)  . 

If  different  groups  of  raters  do  vary  greatly  on 
their  ratings  of  counselor  effectiveness,  it  may  be  that 
effectiveness  is  in  the  "eye  of  the  beholder."     If  this 
is  the  case,  results  of  studies  using  effectiveness 
ratings  of  differing  groups  may  be  not  comparable.  In 
other  words,  the  results  of  studies  on  counselor  effec- 
tiveness could  be  affected  by  the  specific  group  doing 
the  rating.     Conceivably,  researchers  could  be  inad- 
vertently biasing  their  results  by  employing  one  group 
of  raters  instead  of  another. 

On  the  other  hand,  if  the  ratings  of  different 
groups  are  in  agreement  as  to  what  constitutes  effective 


4 


counseling,  this  fact  would  lend  support  to  the  present 
practice  of  comparing  the  findings  of  studies  using  dif- 
ferent groups  of  raters.     Further,  such  agreement  would 
appear  to  justify  counselor  effectiveness  research  which 
employs  one  group  of  raters  when  another  is  unavailable. 

Unfortunately  there  is  no  clear-cut  answer  to  the 
question  of  how  much  counselor  effectiveness  ratings  vary 
as  a  function  of  the  group  membership  of  the  raters.  In- 
formation is  contradictory,  and  often  represents  the 
piecemeal  byproducts  of  more  extensive  investigations. 
Also,  limited  data  exist  regarding  groups  of  raters' 
specific  areas  of  agreement  and  disagreement  within  ef- 
fectiveness rating  scales. 

In  Ivey's   (1971)  recent  book,  Microcounseling ,  Cark- 
huff  stated  that  "all  science  begins  with  the  sensory 
experience  of  the  scientist.     Rather  than  take  it  for 
granted,  the  perception  of  the  perceiver  must  be  studied" 
(p.  viii) .     The  present  study  is  an  attempt  to  move  in 
this  direction  by  investigating  the  relationships  among 
counselor  effectiveness  ratings  of  four  different  groups. 
The  instrument  used  was  the  Counselor  Effectiveness  Scale 
(CES)  Form  2,  a  semantic  differential  measure  appropriate 
for  immediate  use  with  raters  of  varying  sophistication. 

Questions  Posed  by  the  Study 

The  present  study  represents  an  investigation  of  one 
major  research  question  which  was  divided  into  six  sub- 
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questions  concerning  the  relationships  among  counselor 
effectiveness  self-ratings,  peer  ratings,  supervisor 
ratings,  and  client  ratings.     The  questions  under  in- 
vestigation were: 

1.      What  are  the  relationships  among  the  ratings  of 
the  four  groups  of  raters'  on  the  total  score 
of  the  Counselor  Effectiveness  Scale? 

a.  What  is  the  relationship  between  student 
counselor  self-ratings  and  peer  ratings  of 
counselor  effectiveness  on  the  total  score 

"  of  the  Counselor  Effectiveness  Scale? 

b.  What  is  the  relationship  between  student 
counselor  self-ratings  and  supervisor 
ratings  of  counselor  effectiveness  on  the 
total  score  of  the  Counselor  Effectiveness 
Scale? 

c.  What  is  the  relationship  between  student 
counselor  self-ratings  and  client  ratings 
of  counselor  effectiveness  on  the  total 
score  of  the  Counselor  Effectiveness  Scale? 

d.  What  is  the  relationship  between  peer  and 
supervisor  ratings  of  counselor  effective- 
ness on  the  total  score  of  the  Counselor 
Effectiveness  Scale? 

e.  What  is  the  relationship  between  peer  and 
client  ratings  of  counselor  effectiveness 
on  the  total  score  of  the  Counselor 
Effectiveness  Scale? 
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f.      What  is  the  relationship  between  supervisor 
and  client  ratings  of  counselor  effective- 
ness on  the  total  score  of  the  Counselor 
Effectiveness  Scale? 

Definition  of  Terms 

The  following  definitions  are  applicable  in  the  pres- 
ent study: 

Counselor  Effectiveness — In  this  study,  coun- 
selor effectiveness  is  operationally  defined  as  a 
counselor's  rating  on  the  Counselor  Effectiveness 
Scale.  The  CES  was  employed  to  quantify  gradients 
of  counselor  effectiveness,  and  thus  determine  the 
relative  effectiveness  of  a  counselor.  The  higher 
the  counselor's  rating  on  the  CES,  the  more  effec- 
tive he  is  as  Slimed  to  be.  , 

Rater — Any  person  making  quantifiable  judgments 
on  the  CES  as  to  the  relative  effectiveness  of  a 
counselor.     In  this  study,  the  rating  groups  were 
peers,  supervisors,  clients,  and  counselors  them- 
selves. 

Organization  of  the  Remainder  of  the  Study 

A  review  of  the  literature  will  be  found  in  Chap- 
ter II.     Chapter  III  contains  a  discussion  of  the  pro- 
cedures involved  in  the  conduct  of  the  present  study,  as 
well  as  a  discussion  of  the  instrument  used.     Chapter  IV 
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is  devoted  to  the  results  of  the  study  and  Chapter  V  in- 
cludes the  summary,  discussion  and  recommendations  for 
future  research. 


CHAPTER  II 
REVIEW  OF  THE  LITERATURE 


The  present  study  reports  on  the  relationships  found 
among  counselor  effectiveness  ratings  made  by  peers, 
supervisors,  clients,  and  student  counselors  themselves. 
In  order  to  provide  an  adequate  background,  the  present 
chapter  includes  a  review  of  the  literature  on   (a)  rating 
scales;    (b)  the  issue  of  naive  versus  sophisticated  raters 
and  (c)  the  use  of  counselor  effectiveness  ratings  made  by 
supervisors,  counselors,  peers,  clients,  and  coached 
clients . 

Rating  Scales 

Carl  Rogers  and  his  associates  have  been  pioneers  in 
the  use  of  rating  instruments  in  psychotherapy  research. 
Rogers  and  Dymond   (1954)   report  on  some  of  the  first  such 
research  projects.     Rogers  also  conceptualized   (1958)  and 
developed  (1959)  a  "Process  Scale"  for  the  evaluation  of 
seven  parallel  variables.     Studies  investigating  the 
validity  of  this  scale  have  been  encouraging  (Walker, 
Rablen,  and  Rogers,  1960;  Tomlinson  and  Hart,  1962).  Tru- 
ax,  a  student  of  Rogers,  later  developed  the  "Tentative 
Scale  for  the  Measurement  of  Depth  of  Self -Exploration , " 
in  addition  to  other  scales  based  upon  Rogers'  (1957) 
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necessary  and  sufficient  conditions  of  therapeutic  per- 
sonality change.     Truax  and  Carkhuff   (1967)   have  sum- 
marized a  wealth  of  research  based  on  these  scales.  Later, 
Carkhuff   (Carkhuff  and  Berenson,  1967;  Carkhuff,  1969^^].,) 
modified  each  of  Truax 's  therapeutic  condition  scales  to 
include  five  levels  for  each  scale.     Each  point  on  the 
five-point  scales  corresponds  to  a  level  of  therapeutic 
functioning,  quantifying  the  extent  to  which  therapists 
are  judged  to  meet  standardized  criteria.     These  scales 
are  often  used  to  rate  audio-  or  videotaped  excerpts  of 
counseling  interviews.     While  the  reliability  and  validity 
of  Carkhuff 's  scales  are  sometimes  questioned   (Hefele  and 
Hurst,  1972),  a  large  body  of  data  exists  supporting  their 
use   (summarized  in  Carkhuff  and  Berenson,  1967) .  Today, 
Carkhuff ' s  scales  are  probably  the  most  widely  used  rating 
scales  in  studies  of  self-exploration  and  facilitative 
conditions  such  as  empathy,  warmth,  and  genuineness. 

While  both  the  Truax  and  Carkhuff  rating  scales 
have  been  used  extensively,  the  literature  is  also 
replete  with  other,  more  global  scales  of  counselor 
effectiveness   (Walton  and  Sweeney,  1969).     For  example, 
various  types  of  rankings  are  widely  used  as  measures  of 
effective  counseling.     Additional  counselor  effectiveness 
rating  scales  include  the  Counselor  Evaluation  Inventory 
(Linden,  Stone,  and  Shertzer,  1965),  Counselor  Evaluation 
Rating  Scale   (Myrick  and  Kelly,  1971),  Counselor  Poten- 
tial Scale   (Dole,  1964)  ,  Counselor  Practicum  Evaluation 
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Form  (Eberlein  and  Park,  1971)  ,  Self -Anchoring  Scale  of 
Counselor  Effectiveness  (Kilpatrick  and  Cantril,  1960)  , 
Communication  Rating  Scale  (Brams,  1961) ,  Counselor  Eval- 
uation Form  (Faulkenberry ,  1968) ,  and  the  Wisconsin  Rela- 
tionship Orientation  Scale   (VJasson,  1965)  .  Additional 
rating  scales  include  a  free  response,  counselor  percep- 
tion instrument  (Arbuckle,  1956) ,  an  eight-category  rating 
instrument  (Keltz,  1966) ,  a  "sensitivity"  scale  (O'Hern 
and  Arbuckle,  1964) ,  a  sorting  process   (Embree,  1954) , 
sociometric  indices   (Stefflre,  King  and  Leafgren,  1962; 
Gade,  1967)  and  a  variety  of  semantic  differential  rating 
scales   (Shapiro,  1968;  Bartlett  and  Thompson,  1971; 
Ivey,  1971;  Redfering,  1973;  Riese  and  Stoner,  1973). 

For  many  researchers,  the  use  of  such  ratings  solves 
the  problem  of  effectiveness  criteria,  since  most  ratings 
are  readily  accessible,  quantifiable,  and  may  be  used  to 
separate  "good"  counselors  from  "poor"  counselors.  Some 
of  the  above  instruments  were  validated  by  the  extent  to 
which  they  were  found  to  correlate  with  other  measures  of 
counseling  effectiveness   (e.g.,  rankings,  "expert"  judg- 
ments) .     The  validity  of  other  scales  was  inferred  from 
their  reliability.     This  practice  has  been  defended  by  a 
number  of  researchers   (Stefflre,  King,  and  Leafgren, 
1962;  Combs  and  Soper,  1963;  Russo,  Kelz,  and  Hudson, 
1964) .     None  of  the  above  scales  was  directly  validated 
on  client  outcome  criteria. 
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Naive  Versus  Sophisticated  Raters 

An  issue  central  to  counselor  ratings  is  whether 
experienced  counselors  or  supervisors  make  better  raters 
than  do  psychologically  naive  individuals.     While  Rogers 
and  his  associates  initially  believed  that  experienced 
psychotherapists  would  make  the  best  raters,  they  later 
agreed  that  the  use  of  such  a  select  group  would  have 
certain  limitations.     Not  only  would  the  pool  of  raters 
be  restricted,  but  such  sophisticated  raters  may  have  pre- 
conceived biases  as  to  "good  counseling,"  and  thus  be 
influenced  by  a  variety  of  factors  other  than  those 
described  in  the  rating  scales.     Rogers  and  his  asso- 
ciates eventually  used  undergraduate  students  as  raters  in 
the  Wisconsin  Project  (Rogers,  Gendlin,  Kiesler,  and  Truax, 
1967),  reasoning  that  if  therapeutic  conditions  and  pro- 
cess were  truly  objective  and  recognizable,  any  normal 
individual  snould  be  able  to  identify  them. 

Naive  raters  have  been  used  in  a  number  of  other 
studies,  including  those  by  Kiesler,  Mathieu,  and  Klein 
(1964),  Kiesler,  Klein,  and  Mathieu  (1965),  Truax  and 
Carkhuff   (1965),  Truax   (1966),  and  Truax,  War go,  Frank, 
Imber,  Battle,  Hoehn-Sarie,  Nash,  and  Stone   (1966) .  Al- 
so, the  use  of  naive  raters  has  been  supported  by  Arnhoff 
(1954),  Cronbach  (1960),  and  Shapiro  (1968). 

Shapiro   (1968)  compared  the  ratings  of  untrained 
raters  using  a  semantic  differential  rating  scale  with 
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those  of  trained  raters  using  the  Truax  therapeutic  condi- 
tion scales  of  genuineness,  empathy,  and  warmth.  Positive 
correlations  were  found  between  the  ratings  of  the  two 
groups  on  each  of  the  conditions.     These  results  appear 
to  suggest  that  untrained  raters  are  capable  of  differen- 
tiating between  high  and  low  levels  of  facilitative  con- 
ditions.    Similarly,  Par/  (1970)   further  states  that  the 
training  of  raters  for  research  purposes  is  an  uncompli- 
cated procedure  requiring  a  minimum  of  time  and  effort. 

Evidence  also  exists  supporting  the  use  of  more 
sophisticated  raters.     For  example.  Cannon  and  Carkhuff 
(1968)  found  that  rater  level  of  experience  and  the  abili- 
ty to  communicate  therapeutic  conditions  were  significant- 
ly related  to  rater  accuracy,  with  rater  level  of  thera- 
peutic functioning  being  the  more  critical  indicator  of 
rater  accuracy.     Carkhuff   (1969]^)  reports  similar  find- 
ings.    Level  of  rater  functioning,  then,  may  be  an  inter- 
vening variable  influencing  the  accuracy  of  sophisticated 
and  naive  raters.     Other  studies  have  also  suggested  that 
high  functioning  raters  make  more  accurate  ratings  than 
low  functioners   (Carkhuff ,  Kratochvil  and  Friel,  1968; 
Lawlis,  1968).     Similarly,  there  is  evidence  that  low 
functioners  do  not  accurately  rate  their  own  levels  of 
empathy,  regard,  genuineness,  concreteness ,  and  self- 
disclosure   (Burnstein  and  Carkhuff,  1968)  .     These  findings 
support  the  use  of  sophisticated  raters  only  to  the  extent 
that  sophisticated  raters  are  shown  to  be  high  functioners. 
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In  summary,  there  is  evidence  that  the  ratings  of 
psychologically  naive  raters  do  correlate  highly  with 
those  of  trained  raters.     The  evidence,  however,  is  mixed 
and  is  further  complicated  by  research  pointing  to  level 
of  rater  functioning  as  a  critical  factor  in  the  accuracy 
of  ratings. 

Supervisor  Ratings 

Supervisor  ratings  are  generally  considered  to  be 
among  the  most  valid  measures  of  counselor  effectiveness 
(Patterson,  1964) .     The  use  of  such  "expert"  ratings  to 
identify  good  counseling  is  based  on  the  assumption  that 
professional  counselors  can  recognize  various  gradients 
of  effective  counseling  when  they  see  it.     As  Cattell 
stated  in  1903,  "There  is  ...  no  other  criterion  for 
a  man's  work  than  the  estimation  in  which  it  is  held  by 
those  most  competent  to  judge"   (cited  in  Woodworth  and 
Schlosberg,  1954,  p.  57).     Although  the  problem  of  de- 
termining "those  most  competent  to  judge"  remains  un- 
settled, this  rationale  is  generally  employed  to  justify 
the  use  of  supervisor  ratings. 

Supervisor  ratings  are  among  the  most  frequently 
used  criteria  of  effective  counseling.     They  have  been 
used  in  studies  by  Embree   (1954),  McDougall  and  Reitan 
(1961),  Combs  and  Soper   (1963),  Keltz   (1966),  Wicas  and 
Mahan   (1966),  Gade   (1967),  Brown  and  Cannaday  (1969), 
Bishop  (1971),  Friesen  and  Dunning   (1973),  and  numerous 
others.     In  general,  these  studies  report  adequate  inter- 
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rater  reliability.     For  example,  an  interstaff  ranking 
reliability  of  .80  was  found  among  the  five  staff  members 
in  Gade's  1967  study  when  the  coefficient  of  concordance 
was  employed.     Also,  Friesen  and  Dunning  (1973)   found  that 
the  interrater  reliability  of  supervisor  ratings  on  the 
Counselor  Effectiveness  Scale  was  .94,  higher  than  that 
of  both  lay  people  (.92)  and  practicum  students  (.91). 
Similarly,  Falkenberry   (1968)   found  a  higher  interrater 
reliability  for  supervisors  than  for  peers,  counselors, 
and  role-play-clients  in  the  overall  score  and  in  specific 
items  of  the  Counselor  Evaluation  Form. 

Counselor  Self-Ratings 

There  appears,  to  be  conflicting  evidence  as  to  the 
usefulness  of  counselor  self-ratings.     In  support  of  self- 
ratings,  Martin  and  Gazda   (197  0)  concluded  that  counselor 
self-evaluations  of  live  counseling  sessions  had  highly 
significant  concurrent  validity  after  comparing  them  with 
supervisor  evaluations  of  tape  recordings  of  the  same 
sessions.     Martin  and  Gazda  also  support  counselor  self- 
evaluations  on  a  pragmatic  basis.     They  state  that: 

There  are  many  instances  when  counselor  super- 
vision is  impractical  or  impossible.     In  some  college 
and  university  counseling  centers,  for  example,  the 
staff  size  and  counseling  load  limit  supervision, 
and  some  elementary  and  secondary  schools,  vocation- 
al rehabilitation  centers,  and  employment  offices 
are  too  isolated  for  consultation  and  supervision. 
Thus,  self -evaluation  is  needed  to  enable  counselors 
to  evaluate  their  own  efforts,  and  such  a  system 
should  facilitate  personal  and  professional  growth 
(p.  87). 
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Similarly,  many  other  authors  stress  the  importance  of 
a  counselor  being  able  to  examine,  criticize,  and  improve 
upon  his  own  counseling  performance   (Peters  and  Hansen, 
1963;  Patterson,  1964;  Carkhuff  and  Truax,  1965;  Truax, 
1965;  Boy  and  Pine,  1966;  Dreikurs  and  Sonstegard,  1966; 
Miller  and  Getting,  1966). 

Gendlin,  Jenney,  and  Shlien   (1960)   studied  correlates 
of  counselor  judgments  of  success.     They  found  outcome 
ratings  to  be  significantly  correlated  with  measures  of 
the  extent  to  which  counselors  observed  clients   (a)  verbal 
izing  how  their  present  behavior  related  to  more  general 
problems;    (b)  communicating  spontaneously  and  directly; 
and   (c)   talking  about  the  counseling  experience  as  some- 
thing entirely  new.     Research  also  suggests  that  accurate 
counselor  self-evaluations  may  relate  to  measures  of 
client  progress  (Lesser,  1961)  and  client  ratings  of  coun- 
selor variables   (Strupp,  Wallach,  Wogan  and  Jenkins,  1963) 

Several  studies  have  suggested  that  counselors  are 
not  the  best  judges  of  their  own  effectiveness  (Barrett- 
Lennard,  1962;  Rogers  et  al.,  1967;  Truax  and  Carkhuff, 
1967;  Kurtz  and  Grummon,  1972).     In  a  study  by  Kiesler 
(1966),  for  example,  counselors  who  viewed  themselves  as 
offering  relatively  high  levels  of  empathic  understanding, 
congruence,  and  positive  regard  were  perceived  by  clients 
and  by  objective  judges  as  relatively  low  in  these  dimen- 
sions.    Similarly,  Water land   (1965)   found  no  significant 
relationship  between  self-ideal  self  congruence  and  effec- 
tive counseling  performance. 
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In  a  study  by  Poole  (1957) ,  judgments  of  transcript 
readers  were  compared  with  judgments  of  counselors  made 
immediately  before  and  after  counseling  interviews.  Al- 
though counselors  and  readers  generally  agreed  on  the 
client's  problem,  they  disagreed  as  to  what  goals  the 
counselors  appeared  to  have. 

Brams  (1961)  found  a  correlation  of  .73  between  peer 
ratings  and  supervisor  ratings  on  the  Communication  Rating 
Scale.     Self -ratings,  however,  correlated  only  .22  with 
supervisor  ratings  and  .21  with  peer  ratings. 

A  study  by  Brown  and  Cannaday  (1969)  ranked  coun- 
selors according  to  their  own  composite  score  on  the 
Counseling  Evaluation  Inventory   (CEI) ,  according  to  their 
clients'  CEI  score,  and  compared  these  rankings  with  the 
overall  rankings  of  the  supervisor.     The  correlation  be- 
tween client  and  supervisor  rankings  was  .81,  significant 
beyond  the  .01  level  of  significance.     However,  no  sig- 
nificant relationship  was  found  between  counselor  self- 
rankings  and  those  of  either  clients  or  supervisors.  A 
similar  study  by  Bishop  (1971),  however,  seems  to  contra- 
dict thse  results.     In  Bishop's  study,  counselor  self- 
ratings  and  supervisor  ratings  on  the  CEI  were  found  to 
be  significantly  correlated  with  each  other,  but  neither 
was  significantly  correlated  with  client  CEI  ratings. 

Several  factors  may  help  explain  the  apparent  contra- 
diction in  research  on  counselor  self -ratings .     It  may  be 
that  agreement  between  counselor  self -ratings  and  other 
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kinds  of  ratings  improves  over  time  (Carkhuff,  Kratochvil, 
and  Friel,  1968;  MacGuffie,  Janzen,  and  McPhee,  1970),  or 
that  agreement  varies  as  a  function  of  the  amount  of  inter- 
action among  rating  groups   (Engle  and  Betz,  1971).  Also, 
agreement  may  vary  as  a  function  of  the  facilitative  con- 
ditions offered  by  the  counselor   (Burnstein  and  Carkhuff, 
1968).     There  is  no  information,  however,  regarding  the 
extent  to  which  these  or  other  factors  influence  self- 
ratings  of  counselor  effectiveness.     Likewise,  confusion 
remains  regarding  the  relationships  between  counselor 
self-ratings  and  the  counselor  effectiveness  ratings  of 
other  groups. 

Peer  Ratings 

Peer  ratings  are  being  used  increasingly  in  counselor 
education.     McDougal  and  Reitan  (1961)   list  the  following 
advantages  of  employing  peer  evaluations  with  counselors- 
in-training: 

1.  It  [the  use  of  peer  evaluations]  may  afford  a 
means  of  revealing  important  behaviors  which 
might  be  overlooked  or  obscured  in  conventional 
evaluative  procedures. 

2.  Used  and  interpreted  with  discretion,  it  gives 
the  student  an  unusual  opportunity  to  evaluate 
himself  in  the  light  of  peer  appraisal. 

3.  Used  periodically  throughout  a  training  period, 
it  allows  for  an  evaluation  of  trainee  progress, 
both  to  self  and  instructor. 

4.  It  may  afford  a  means  of  helping  to  remove  psy- 
chological barriers  which  prevent  realistic 
self-appraisal . 

5.  It  may  provide  the  instructor  with  an  opportunity 
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for  examining  his  own  bias  in  making  judgments 
of  students'  behavior  (p.  76). 

Finally,  as  a  counselor  learns  to  identify  effective  and 
ineffective  counseling  behaviors,  he  may  begin  to  in- 
crease his  own  self-corrective  capabilities   (Friesen  and 
Dunning,  1973). 

A  good  deal  of  research  also  supports  the  use  of  peer 
ratings  as  criteria  for  counseling  effectiveness.  Gen- 
erally this  research  points  to  high  peer  interrater  relia- 
bility and  positive  correlations  between  peer  ratings  and 
the  "accepted  criteria"  of  supervisor  ratings.  For 
example,  Friesen  and  Dunning   (1973)   suggested  the  validity 
of  peer  ratings  on  the  basis  of  a  rank  order  correlation 
of  .88  between  peer  and  supervisor  ratings  on  the  Coun- 
selor Effectiveness  Scale.     The  interrater  reliability  for 
the  peers  in  this  study  was  above  .90.    A  similar  study 
by  McDougall  and  Reitan  (1961)  used  peer  ratings  which  re- 
quired the  forcing  of  classmates '   judgments  into  quarter 
cells.     Using  a  rank  order  correlation,  the  investigators 
found  that  peer  ratings  positively  correlated  .61  with 
supervisor  ratings.     Dilley  (1964)  found  a  Pearson  Product 
Moment  Correlation  of  .65  between  peer  rankings  and  super- 
visor nominations  of  "most  effective"  and  "least  effec- 
tive" counselors.     Also,  a  study  by  Embree   (1954)  found 
that  a  sorting  process  used  by  supervisors  and  rankings 
of  counselor  effectiveness  by  peers  were  generally  con- 
sistent with  one  another. 

Peer  and  supervisor  judgments  have  been  found  to 
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correlate  higher  in  studies  using  NDEA  Institute  groups 
(McDougal  and  Reitan,  1961;  Stefflre,  King,  and  Leafgren, 
1962;  Dilley,  1964)   than  in  comparable  studies  conducted 
within  regular  counselor  education  programs  (Garrison, 
1967;  Mendoza,  1968).     For  example,  the  correlations  in 
the  Institute  studies  have  been  positive  and  significant, 
and  have  ranged  from  .60  to  .90,  while  Garrison's  study 
reported  a  much  lower  correlation  of  .42  and  Mendoza  re- 
ported .56.     In  order  to  investigate  this  apparent  dis- 
crepancy, Engle  and  Betz   (1971)   examined  the  relationship 
between  supervisor  and  peer  ratings  in  an  NDEA  Institute 
and  in  a  regular  counselor  education  program.     The  results 
of  their  investigation  indicated  that  NDEA  Institute  super- 
visors and  students  had  significantly  higher  positive 
agreement  (.77)  on  who  was  a  good  counselor  than  super- 
visors and  counselor  trainees  in  a  regular  counselor  edu- 
cation program  (.46).     The  authors  hypothesized  that  the 
critical  difference  in  the  relationships  was  due  to  the 
greater  contact  among  Institute  members,  which  resulted 
in  a  better  knowledge  of  one  another's  counseling  ability. 

In  summary,  peer  ratings  of  counseling  effectiveness 
typically  have  adequate  interrater  reliability  and  have 
been  shown  to  correlate  consistently  with  supervisor 
ratings,  although  it  appears  that  the  extent  of  peer- 
supervisor  agreement  varies  as  a  function  of  the  type  of 
counseling  program.     While  a  high  correlation  between 
peer  and  supervisor  ratings  may  indeed  imply  the  validity 
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of  peer  ratings,  a  word  of  caution  is  in  order.     As  Patter 
son  (1967)  has  stated,  the  relationship  between  peer  and 
supervisor  ratings  of  counselor  effectiveness  "may  be  due 
to  the  sharing  of  common  biases,  and  may  thus  represent 
reliability  rather  than  validity"   (p.  79) . 

Client  Ratings 

Many  authors  agree  that  client  ratings  are  of  value 
in  determining  counselor  effectiveness   (Shoben,  1953; 
Form,  1955;  Arbuckle,  1956;  Patterson,  1958;  Goodstein 
and  Grigg,  1957,  1959;  Pohlraan  and  Robinson,  1960;  Grigg, 
1961;  Pohlman,  1961;  Stablein,  1962;  Mueller,  Gatsch,  and 
Ralston,  1963;  Linden,  Stone,  and  Shertzer,  1965; 
Severinsen,  1966;  Rosen,  1967;  and  Thompson  and  Miller, 
1970) .     In  addition,  several  studies  have  demonstrated 
that  client  rating  instruments  such  as  the  Counselor 
Evaluation  Inventory   (CEI)  correlate  highly  with  super- 
visor rankings,  and  hence  may  be  measuring  criteria  that 
supervisors  think  important   (Covell,  1956;  Poole,  1957; 
Anderson  and  Anderson,  1962).     Using  the  CEI,  Rickabaugh, 
Heaps,  and  Finley  (1972)  also  found  that  clients'  percep- 
tions of  counselor  comfort  were  significantly  related  to 
another  measure  of  counselor  effectiveness,  that  of 
client  academic  grade  improvement.     In  another  study, 
clients  using  the  CEI  have  been  found  to  give  consistently 
higher  ratings  to  counselors  with  practicum  experience 
than  to  counselors  without  a  practicum  (Pfeifle,  1971). 
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-  -    '^Client  ratings  also  have  been  used  to  investigate 
the  therapeutic  conditions  of  empathy,  warmth,  and  genuine 
ftess  of f ered  by  the  counselor  (Truax  and  Carkhuff,  1967). 
Barrett-Lennard  (1962)  was  one  of  the  first  to  measure 
the- client's  perceptions  of  the  counselor's  therapeutic 
conditions.     However,  the  usefulness  of  such  client  per- 
ceptions in  predicting  objective  ratings  and/or  outcomes 
is  unclear;  studies  that  have  employed  relationship  in- 
ventories often  have  had  differing  results.  Positive 
findings  have  been  reported  with  juvenile  delinquents 
^Truax,  Wargo,  Carkhuff,  Tunnel 1,  and  Glenn,  1966) , 
outpatients  (Truax,  Wargo,  Tunnell,  and  Glenn,  1966)  , 
schizophrenics   (Rogers    et  al.,  1967),  psychoneurotics 
^Barrett-Lennard,  1962),  and  undergraduate  clients  at  a 
university  counseling  center   (Kurtz  and  Gummon,  1972) . 
Studies  with  differing  findings  include  those  using 
schizophrenic  patients   (Truax,  1962^^]^),  high  school 
students  in  group  counseling   (Hansen,  Moore,  and  Cark- 
huff, 1968) ,  hospitalized  mental  patients  and  institu- 
■trl?>fiaiized  male  juvenile  delinquents  in  group  psycho- 
therapy (Truax,  1966),  and  a  nationwide  sample  of  clients 
unidentified  by  name,  age  or  institutional  affiliation 
(Kaul,  Kaul,  and  Bednar,  1973). 

Criticism  has  sometimes  been  leveled  against  client 
ratings  because  of  their  subjectivity  and  tendency  toward 
bias   (Pohlman,  1961,  1964;  Patterson,  1958,  1959;  Rosen, 
1967).     Unrealistic  client  expectations  have  also  been 
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said  to  affect  client  satisfaction  (Patterson,  1958;  Sev- 
erinsen,  1966) .     A  study  by  Pohlman  and  Robinson  (1960) 
seems  to  confirm  the  view  that  clients  do  have  definite 
counselor  preferences.     In  this  study,  counselor  behaviors 
were  presented  verbally  to  a  psychology  class,  and  stu- 
dents rated  their  degree  of  annoyance  or  liking.     Of  the 
92  statements,  73  received  mean  ratings  significantly 
different  from  the  mean  of  neutrality.    Also,  sex  group 
differences  occurred  on  26  items,  all  in  the  direction 
of  greater  female  annoyance.     Moreover,  Rosen's  (1967) 
review  of  the  literature  on  client  preferences  lists  a 
number  of  other  client  preferences  which  may  influence  a 
client's  rating  of  counselor  effectiveness. 

As  for  client  preferences  for  counseling  approaches, 
Barahal  (1950)  found  that  clients  receiving  "client- 
centered  counseling"  reported  greater  satisfaction  with 
the  results  of  counseling  than  those  counseled  by  "tradi- 
tional" techniques.     The  findings  of  this  study  may  be 
misleading,  however,  since  client  preferences  for  various 
counseling  approaches  appear  to  be  related  to  the  type  of 
problem  experienced  by  the  client   (Berdin,  1955) ,  the 
personality  of  the  client  (Sonne  and  Goldman,  1957),  the 
identity  of  the  counselor   (Grigg  and  Goodstein,  1957),  and 
counselor  activity  (Forgy  and  Black,  1954),  as  well  as  to 
a  number  of  other  possible  variables. 

Patterson   (1959),  among  others,  has  pointed  out  that 
while  clients  have  definite  preferences  regarding  the 
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counseling  relationship,  the  counselor  should  not  neces- 
sarily try  to  conform  to  what  the  client  wants.  For 
example,  while  the  client  may  like  to  be  told  what  to  do 
by  a  "mothering"  counselor,  such  a  relationship  may  not 
be  in  the  client's  best  interest.     This  type  of  situation 
has  specific  implications  for  client  ratings,  since  a 
client  may  rate  a  counselor  positively  when  the  counselor 
is  actually  compounding  the  client's  problem.  Moreover, 
various  studies  have  suggested  that  the  client's  percep- 
tion of  the  counselor's  facilitative  conditions  may  be 
distorted  (Truax,  1961;  Burnstein  and  Carkhuff,  1968). 

In  conclusion,  there  is  substantial  research  support- 
ing the  use  of  client  ratings  of  counselor  effectiveness. 
However,  client  satisfaction  is  a  complex  variable,  de- 
pendent upon  client  preferences,  expectations,  and  numer- 
ous other  factors.     Therefore,  client  ratings  of  coun- 
selor effectiveness  should  be  employed  and  integrated 
with  caution.     This  is  why  Goodstein  and  Grigg  (1959) 
suggest  multiple  criterion  measures  of  counselor  effec- 
tiveness.    They  state  that  "client  satisfaction  is  one 
important  factor  in  any  multifactor  approach  to  the 
problem  of  effectiveness  in  counseling"   (p.  23). 

Coached  Client  Ratings 

Coached  clients  are  sometimes  used  in  counselor 
effectiveness  research  to  control  for  possible  client 
bias  and  subjectivity.     Mcllvaine   (1972)   states  that: 
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If  client  ratings  are  used  to  deteirmine  the 
counseling  effectiveness  of  counselor  trainees, 
then  it  is  important  to  implement  procedures  that 
will  insure  that  these  client  ratings  are  free  from 
common  rating  error  and  rater  bias.     The  coached- 
client  methodology  .   .   .  appears  to  enhance  this 
needed  client  objectivity  in  the  rating  of  the  coun- 
seling effectiveness  of  counselor  trainees  (p.  128) . 

To  examine  the  relative  accuracy  of  coached  client 
ratings,  Mcllvaine   (1972)  compared  the  Counseling  Evalua- 
tion Inventory  ratings  of  two  coached  clients  trained  in 
objective  rating  techniques  with  those  of  two  noncoached 
clients.     The  CEI  scores  of  the  coached  clients  agreed 
more  closely  with  the  counseling  effectiveness  ratings  of 
practicum  supervisors  than  did  the  scores  of  noncoached 
clients. 

The  behavior  of  coached  clients  is  usually  held  con- 
stant by  objectively  defined  roles.     In  this  way,  a 
number  of  counselors  may  be  rated  under  similar  circum- 
stances  (Whitely  and  Jakubowski,  1969) ,  or  counselors 
may  be  rated  under  systematically  varied  client  condi- 
tions  (Heller,  Myer,  and  Kline,  1963).     Generally,  there 
is  agreement  that  coached  clients  have  a  variety  of  po- 
tential uses  in  counselor  training  and  counselor  effec- 
tiveness research  (Heller,  Myer,  and  Kline,  1963;  Keltz, 
1966;  Carkhuff  and  Alexik,  1967;  Pierce,  Carkhuff,  and 
Berenson,  1967;  Miller  and  Befus,  1968;  Whitely  and 
Sprandel,  1969;  Whitely  and  Jakubowski,  1969;  Mcllvaine, 
1972). 
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Suimnary 

Ratings  by  counselors,  peers,  supervisors,  clients 
and  coached  clients  have  been  used  extensively  in  re- 
search as  criteria  of  counselor  effectiveness.  While 
some  data  support  the  use  of  ratings  made  by  each  of 
these  groups,  conflicting  data  also  exist.     In  addition, 
it  is  unclear  to  what  extent  these  different  rating 
groups  agree  on  ratings  of  counselor  effectiveness.  In- 
formation is  incomplete  and  often  contradictory.  Clearly, 
there  is  a  need  for  more  extensive  research  regarding  the 
comparability  of  effectiveness  ratings  made  by  members  of 
different  groups. 


CHAPTER  III 
PROCEDURES 


The  present  study  is  an  attempt  to  investigate  the 
extent  to  which  counselor  effectiveness  ratings  vary  as 
a  function  of  the  group  membership  of  the  raters.  This 
study  examined  the  consistency  of  effectiveness  ratings 
across  four  rating  groups. 

Subjects  and  Their  Selection 

The  rating  groups  employed  in  the  present  study  were 
University  of  Florida  student  counselors,  their  peers, 
their  supervisors,  and  their  clients.     These  groups  and 
their  selection  are  described  below:  ^  , 

1.       Student  Counselors — The  counselor  self -rating 
group  was  composed  of  graduate  students  in  the 
Department  of  Counselor  Education  at  the  Uni- 
versity of  Florida.    At  the  time  of  the  study 
these  student  counselors  were  involved  in  a 
practical  counseling  experience,  either  a  prac- 
ticum  or  an  internship,  through  the  Department 
of  Counselor  Education.     The  agencies  in  which 
these  student  counselors  were  working,  along  with 
the  number  of  student  counselor  subjects  in  each 
agency,  are  presented  in  Table  1.  Graduate 
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TABLE  1 

Distribution  of  Student  Counselor  Subjects 
by  Practicum  or  Internsip  Agency 


Number  of  Student 

Counselor  Subjects  Agency 


5  Group  Leader,  Santa  Fe  Community 

College 

5  Psychological  and  Vocational  Counsel- 

ing Center,  University  of  Florida 

3  North  Central  Florida  Community 

Mental  Health  Clinic 

2  "  CREST  (Clinical  Regional  Support 

Teams) ,  a  Community-Based  Juvenile 
Delinquency  Project 

2  ,      Human  Service  Aide  Program,  Santa 

Fe  Community  College 

2  Student  Mental  Health,  University 

of  Florida 

2  Suicide  and  Crisis  Intervention 

^       .  Center,  Gainesville 

1  Bradford-Union  County  Guidance  Clinic 

1  Child,  Youth  and  Family  Center,  a 

Division  of  North  Central  Florida 
Community  Mental  Health 

1  Division  of  Family  Services, 

Gainesville 

1  Florida  Correctional  Institute, 

Lowell 

1  Reading  and  Study  Skills  Center, 

University  of  Florida 

1  Special  Services,  University  of 

Florida 
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student  counselors  working  in  elementary  schools 
were  not  included  in  the  present  study  because 
of  the  difficulty  in  securing  client  ratings 
from  elementary  school  children. 

The  number  of  student  counselors  that  each 
supervisor  sees  in  individual  supervision  varies 
widely.     In  this  study,  one  counselor  was  ran- 
domly selected  from  each  supervisor's  pool  of 
supervisees.     This  stratified  random  sampling 
of  student  counselors  was  employed  in  order  to 
obtain  independency  of  measurements,  as  this 
procedure  makes  certain  that  no  two  counselors 
are  rated  by  the  same  supervisor.     Thus,  each 
of  27  different  counselors  was  rated  by  27 
separate  supervisors  on  the  CES. 
Peers — The  peers  in  this  study  were  also  grad- 
uate students  at  the  University  of  Florida. 
Like  the  counselor  subjects,  the  peers  were 
involved  in  either  a  practicum  or  internship. 

Peers  were  randomly  selected  from  those 
individuals  attending  each  particular  counselor 
subject's  group  supervision  session.     In  some 
cases,  counselor  education  graduate  student 
counselors  attend  agency  staff  meetings  instead 
of  departmental  group  supervision.     In  such 
cases,  the  corresponding  peer  counselor  was 
randomly  selected  from  those  counselor  education 
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students  in  the  counselor  subject's  work  setting. 
In  agencies  where  no  counselor  education  peers 
were  available,  clinical  psychology  graduate 
students  were  randomly  selected  as  peers.  The 
peer  group  included  20  counselor  education 
graduate  students  and  seven  clinical  psychology 
graduate  students.     All  peers  were  either  in 
group  supervision  or  in  a  work  setting  with  the 
student  counselors  for  a  minimum  of  four  weeks. 

3.  Supervisors — The  supervisors  in  the  present 
study  were  all  from  the  Department  of  Counselor 
Education  at  the  University  of  Florida  or 

from  the  student  counselor's  work  setting.  All 
had  been  individual  supervisors  to  the  student 
counselor  subjects  for  at  least  four  weeks. 
The  majority  of  the  supervisors  held  doctorates 
in  fields  such  as  Counselor  Education  and  Clin- 
ical Psychology.     Five  were  advanced  doctoral 
students  in  the  Department  of  Counselor  Educa- 
tion, all  of  whom  had  had  several  years  of 
counseling  experience. 

4.  Clients — The  clients  in  this  study  were  indi- 
viduals whom  the  counselor  subjects  saw  in 
counseling  for  a  predetermined  week.  Clients 
must  have  been  seen  by  a  counselor  subject  for 
a  minimum  of  two  sessions  to  be  included  in 
the  study. 


The  mean  age  and  number  of  subjects  by  sex  for  each 
rating  group  are  presented  in  Table  2 . 

Counselor  Effectiveness  Scale 

Various  forms  of  semantic  differential  instruments 
have  been  used  as  rating  scales  in  counseling  research 
(Shapiro,  1968;  Bartlett  and  Thompson,  1971;  Ivey,  1971; 
Redfering,  1973;  Riese  and  Stoner,  1973;  Schmidt,  1974). 
The  rating  scale  used  in  the  present  study  was  the  Coun- 
selor Effectiveness  Scale  (CES) ,  Form  2,  a  semantic  dif- 
ferential instrument  developed  by  Ivey   (1971)    (See  Appen- 
dix A.).     This  scale  contains  25  items,  with  each  item 
having  seven  steps  between  a  positive  counselor  trait 
(e.g.,  helpful)  and  a  negative  counselor  trait  (e.g., 
unhelpful).     For  scoring  purposes,  the  steps  between  the 
traits  were  assigned  numerical  scores  with  a  score  of 
seven  for  the  step  closest  to  the  most  positive  trait 
and  a  score  of  one  for  the  step  closest  to  the  least  posi- 
tive trait. 

The  CES  was  originally  constructed  from  a  randomly 
ordered  initial  item  pool  of  93  semantic  differential 
adjective  pairs.     Using  these  items,  two  graduate  psy- 
chology classes  rated  two  videotaped  coached  counselors, 
one  displaying  "desirable"  counselor  behavior  and  the 
other  portraying  "undesirable"  or  "ineffective"  tech- 
niques.    Means,  standard  deviations,   standard  errors, 
and  confidence  limits   (.05  and  .01)  were  then  computed 
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TABLE  2 

Mean  Age  and  Number  of  Subjects 
by  Sex  for  Each  Rating  Group 


Rating 

Male 

Female  Total 

Group 

N  Mean  Age 

N   Mean  Age     N   Mean  Age 

Student  Counselors 

9 

27 

.3 

18 

30 

.2 

27 

29 

.2 

Peers 

14 

30 

.1 

13 

26 

.4 

27 

28 

.3 

Supervisors 

17 

38 

.6 

10 

39. 

6 

27 

39 

0 

Clients 

8 

23. 

3 

19 

23. 

4 

27 

23. 

4 

32 


for  the  ratings  of  each  counselor.     The  standard  error 
of  each  item  served  as  a  measure  of  its  reliability, 
while  the  difference  between  confidence  limits  of  the 
positive  and  negative  models  served  as  a  measure  of  item 
validity.     The  two  parallel  forms  of  25  items  were  con- 
structed after  those  items  were  selected  which  both  re- 
flected high  agreement  among  raters   (low  standard  error) 
and  which  were  able  to  discriminate  between  counselors 
(wide  range  between  confidence  limits  for  positive  versus 
negative  model)    (Ivey,  1971,  p.  181) . 

The  CES  was  chosen  for  the  present  study  upon  review 
of  the  following  criteria  proposed  by  Osgood   (1959)  : 

1.  Objectivity — The  CES  is  objective  in  that  the 
'  .'ly         data  may  be  made  explicit,  quantified,  and 

'  ■''         hence  may  be  replicated. 

2.  Reliability — Interrater  reliability  was  ini- 
tially computed  by  means  of  Kendall's  Coeffi- 
cient of  Concordance  in  order  to  obtain  a 
measure  of  agreement  among  seven  judges  (Ivey, 
1971) .     Each  judge  rated  a  videotape  model  of 
counseling  on  50  observations   (25  on  Form  1 
and  25  on  Form  2) .     When  each  of  the  adjective 
continuums  was  compared  separately  rather  than 
by  the  total  effectiveness  score,  a  value  of 
.37  was  obtained.     This  correlation  coefficient 
is  significant  at  the  .001  level  of  confidence. 

When  Friesen  and  Dunning   (1973)  computed 
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interrater  reliability  on  the  CES  using  total 
scores  rather  than  item  scores,  they  found 
correlations  above  .90  for  each  group  of  raters. 
Validity — An  estimate  of  the  validity  of  the 
CES  was  initially  based  on  the  extent  to  which 
naive  raters  (N=18)  could  differentiate  between 
a  videotape  of  a  rationally  defined  "good" 
counseling  model  and  a  videotape  of  a  ration- 
ally defined  "poor"  counseling  model.  The 
mean  rating  of  the  good  model  was  131.22  for 
Form  1  and  134.44  for  Form  2  while  the  poor 
model  received  ratings  of  66.25  on  Form  1  and 
65.00  on  Form  2.     The  difference  between  the 
good  and  bad  model  for  each  form  was  signifi- 
cant beyond  the  .001  level  of  confidence. 
Form  2  was  chosen  for  the  present  study  because 
of  its  greater  range  of  differentiation. 
Sensitivity — Osgood  and  his  associates  (1957) 
describe  a  great  deal  of  research  reflecting 
the  fine  discriminative  qualities  of  the  seman- 
tic differential.     While  these  qualities  may 
or  may  not  be  related  to  validity,  they  seem 
to  indicate  that  the  semantic  differential 
generally  is  able  to  discriminate  fine  dif- 
ferences in  ratings.     More  specifically,  re- 
search seems  to  indicate  that  the  CES  is  a 
sensitive  rating  scale  of  counseling  effective- 


34 


ness  (Ivey,  1971;  Ivey,  personal  communication, 
1974)  . 

5.      Comparability — The  CES  is  easily  understood 
and  thus  may  be  used  by  groups  of  raters  of 
varying  degrees  of  sophistication.  Relation- 
ships among  such  groups  of  raters  may  therefore 
be  compared  systematically. 

Administration  of  the  Counselor  Effectiveness  Scale 

The  present  researcher  administered  the  CES  to  each 
of  27  supervisors.     Each  supervisor  then  rated  a  randomly 
selected  counselor  supervisee  and  returned  the  rating 
sheet  to  the  researcher. 

Each  counselor  subject  was  asked  by  the  present 
researcher  to  provide  demographic  data  (see  Appendix  B) 
and  to  rate  himself  on  the  CES  during  his  group  super- 
vision session.     Also,  at  each  counselor  subject's  group 
supervision  session  a  randomly  selected  peer  graduate 
student  counselor  rated  the  counselor  subject  on  the  CES. 
No  peer  rated  more  than  one  counselor  subject.     In  this 
way,  one  separate  peer  and  one  self-rating  were  obtained 
on  each  of  the  27  counselor  subjects.     If  any  counselor 
subject  did  not  attend  a  formal  group  supervision  session, 
he  and  a  randomly  selected  peer  from  his  work  setting  were 
administered  the  CES  at  the  work  setting. 

Each  counselor  subject  had  each  client  that  he  saw 
for  a  predetermined  week  rate  him  on  the  CES.     This  scale 
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was  administered  to  the  client  by  his  counselor  immediate- 
ly after  the  counseling  session.     The  client  completed 
the  scale  in  private  and  sealed  it  in  a  stamped  envelope 
addressed  to  the  present  researcher.     The  client  then 
either  gave  the  envelope  to  the  secretary  at  that  agency 
to  mail  or  mailed  it  himself.     One  randomly  selected 
client  rating  for  each  student  counselor  was  used  in  the 
data  analyses. 

Limitations  of  the  Study 

Since  this  study  focused  specifically  on  University 
of  Florida  counselor  education  students,  their  peers, 
clients,  and  supervisors,  generalizability  is  limited. 
In  addition,  the  fact  that  counselor  education  students 
who  worked  in  elementary  schools  were  excluded  from  this 
study  further  limits  the  generalizability  of  results. 

The  Counselor  Effectiveness  Scale,  a  global  rating 
of  counselor  effectiveness,  was  employed  in  the  present 
study.     While  specific  measures  of  counselor  effective- 
ness such  as  empathy,  warmth,  genuineness,  self -explora- 
tion, self -disclosure,  openness,  creativity,  or  self- 
actualization  may  be  partially  reflected  in  this  global 
rating,  these  conditions  were  not  examined  directly. 
Therefore,  one  should  not  generalize  beyond  the  specific 
instrument  used  in  the  present  study. 

The  CES,  like  most  rating  scales  of  counselor  effec- 
tiveness, was  not  validated  on  client  outcome  measures. 
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Therefore,  this  study  did  not  directly  investigate  the 
validity  of  counselor  effectiveness  ratings.  Instead, 
this  study  examined  reliability  by  determining  correla- 
tions between  various  groups'  ratings  on  the  CES. 

Null  Hypotheses 

One  major  null  hypothesis  was  divided  into  six  sub- 
hypotheses  to  be  tested  in  the  present  study: 

1.      There  are  no  significant  correlations  [p<..01] 
among  the  four  groups  of  raters '  counselor 
effectiveness  ratings  on  the  total  score  of 
the  Counselor  Effectiveness  Scale. 

a.  There  is  no  significant  correlation  be- 
tween student  counselor  and  peer  ratings 
of  counselor  effectiveness  on  the  total 

K  ^    -    ,   '.  ,     score  of  the  Counselor  Effectiveness 
Scale. 

b.  There  is  no  significant  correlation  be- 
tween student  counselor  and  supervisor 
ratings  of  counselor  effectiveness  on  the 
total  score  of  the  Counselor  Effective- 
ness Scale. 

c.  There  is  no  significant  correlation  be- 
tween student  counselor  and  client  ratings 
of  counselor  effectiveness  on  the  total 
score  of  the  Counselor  Effectiveness  Scale. 

d.  There  is  no  significant  correlation  between 
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peer  and  supervisor  ratings  of  counselor 
effectiveness  on  the  total  score  of  the 
Counselor  Effectiveness  Scale. 

e.  There  is  no  significant  correlation  be- 
tween peer  and  client  ratings  of  counselor 
effectiveness  on  the  total  score  of  the 
Counselor  Effectiveness  Scale. 

f.  There  is  no  significant  correlation  be- 
tween supervisor  and  client  ratings  of 
counselor  effectiveness  on  the  total  score 
of  the  Counselor  Effectiveness  Scale. 

Analysis  of  the  Data 

The  present  study  investigated  the  comparability  of 
counselor  effectiveness  ratings  made  by  four  different 
groups.     More  specifically,  this  study  examined  the  rela- 
tionships among  student  counselors'  counselor  effective- 
ness self -ratings,  peer  ratings,  supervisor  ratings,  and 
client  ratings  on  the  Counselor  Effectiveness  Scale, 
Form  2.     The  statistical  analyses  of  the  relationships 
among  group  ratings  were  performed  with  the  use  of  the 
Pearson  product-moment  correlation.     This  statistic  was 
also  used  to  investigate  the  degree  of  interrelationship 
of  group  ratings  on  the  specific  items  within  the  Coun- 
selor Effectiveness  Scale.     A  confidence  level  of  .01 
was  set  for  the  total  score  correlations,  as  this  level 
was  judged  by  the  researcher  to  account  for  an  acceptable 
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degree  of  rating  variance.     A  one-way  analysis  of  variance 
was  employed  to  test  whether  the  mean  ratings  for  each 
rating  group  differed  significantly  from  those  of  the 
other  rating  groups.     A  confidence  level  of  .05  was  used 
to  determine  significant  differences  in  group  ratings. 

The  means  and  standard  deviations  for  each  rating 
group  on  the  CES  were  computed  through  the  use  of  the 
computer  program  SPSSH,  version  V600.     The  computer  pro- 
gram used  for  the  Pearson  product-moment  correlations 
was  also  the  SPSSH,  version  V600.     The  one-way  analysis 
of  variance  was  computed  by  means  of  the  computer  program 
UFSPL  102.     These  computations  were  performed  with  an 
IBM  360,  modal  40  computer  at  the  University  of  Florida 
Computer  Center. 


CHAPTER  IV 


RESULTS 

The  present  study  was  conducted  to  examine  the  con- 
sistency of  counselor  effectiveness  ratings  across  four 
different  rating  groups.     Each  null  hypothesis  originally 
stated  in  Chapter  III  is  presented  below,  along  with  the 
data  used  in  testing  the  hypothesis.     A  summary,  discus- 
sion, and  recommendations  for  future  research  are  pre- 
sented in  Chapter  V. 

Results  in  Relation  to  Specific  Hypotheses 

Null  Hypothesis  la 

There  is  no  significant  correlation  [.01]  between  student 
counselor  and  peer  ratings  of  counselor  effectiveness  on 
the  total  score  of  the  Counselor  Effectiveness  Scale. 

An  intercorrelation  matrix  for  this  and  the  remain- 
ing five  null  hypotheses  is  presented  in  Table  3.  For  a 
correlation  to  be  significant  at  the  .01  level  of  confi- 
dence (df=26) ,  it  must  have  been  .48  or  above.  Since  .20 
was  the  correlation  found  between  the  CES  total  scores 
for  student  counselors  and  peers.  Null  Hypothesis  la  was 
not  rejected. 
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TABLE  3 

Intercorrelation  Matrix  on 
Counselor  Effectiveness  Scale,  Form  2,  Total  Score 


Student 

Counselors    Peers     Supervisors  Clients 


Student  Counselors  .20              .08  .32 

Peers  .26  .23 

Supervisors  '  .12 
Clients 
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Null  Hypothesis  lb 

There  is  no  significant  correlation  between  student  coun- 
selor and  supervisor  ratings  of  counselor  effectiveness 
on  the  total  score  of  the  Counselor  Effectiveness  Scale. 

As  shown  in  Table  3,  a  correlation  of  .08,  not  sig- 
nificant at  the  .01  level,  was  found  between  the  CES 
total  scores  for  student  counselors  and  supervisors. 
Accordingly,  Null  Hypothesis  lb  was  not  rejected. 

Null  Hypothesis  Ic 

There  is  no  significant  correlation  between  student 
counselor  and  client  ratings  of  counselor  effectiveness 
on  the  total  score  of  the  Counselor  Effectiveness  Scale. 

A  correlation  of  .32  was  found  between  the  CES  total 
scores  for  student  counselors  and  clients.     Since  this 
correlation  did  not  reach  the  critical  level  of  signifi- 
cance. Null  Hypothesis  Ic  was  not  rejected. 

Null  Hypothesis  Id    ^ ;  - 

There  is  no  significant  correlation  between  peer  and 
supervisor  ratings  of  counselor  effectiveness  on  the 
total  score  of  the  Counselor  Effectiveness  Scale. 

The  correlation  between  the  CES  total  scores  for 
peers  and  supervisors  was  found  to  be  .26,  not  signifi- 
cant at  the  .01  level  of  confidence.     Null  Hypothesis  Id 
was  therefore  not  rejected. 

Null  Hypothesis  le 

There  is  no  significant  correlation  between  peer  and 
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client  ratings  of  counselor  effectiveness  on  the  total 
score  of  the  Counselor  Effectiveness  Scale. 

Since  a  correlation  of  .23,  not  significant  at  the 
.01  level,  was  found  between  the  CES  total  scores  for 
peers  and  clients.  Null  Hypothesis  le  was  not  rejected. 

Null  Hypothesis  If 

There  is  no  significant  correlation  between  supervisor 
and  client  ratings  of  counselor  effectiveness  on  the 
total  score  of  the  Counselor  Effectiveness  Scale. 

A  nonsignificant  correlation  of  .12  was  found  be- 
tween the  CES  total  scores  for  supervisors  and  clients. 
Accordingly,  Null  Hypothesis  If  was  not  rejected. 

Other  Analyses 

The  pairwise  Pearson  product-moment  correlations  for 
each  adjective  pair  on  the  Counselor  Effectiveness  Scale 
are  presented  in  Table  4 .     As  shown  in  Table  4 ,  the  only 
correlation  that  was  significant  at  the  .01  level  of 
confidence  was  the  correlation  of  .59  between  the  ratings 
of  student  counselors  and  clients  on  the  item  "strong- 
weak."    The  eight  correlations  significant  at  the  .05 
level  of  confidence  were:      (1)  on  the  item  "skillful- 
clumsy,"  a  correlation  of   .39  between  ratings  of  student 
counselors  and  supervisors,    (2)  on  the  item  "deep-shallow, 
a  correlation  of  .45  between  ratings  of  student  counselors 
and  peers,  and  a  correlation  of  .42  between  ratings  of 
student  counselors  and  supervisors,    (3)  on  the  item 
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"socially  inept-socially  adept/'  a  correlation  of  .41  be- 
tween the  ratings  of  student  counselors  and  peers,  (4) 
on  the  item  "friendly-hostile,"  a  correlation  of  .40  be- 
tween ratings  of  student  counselors  and  clients,  and  a 
correlation  of  .44  between  ratings  of  peer  and  clients, 
(5)  on  the  item  "erratic-stable,"  a  correlation  of  .47 
between  the  ratings  of  peers  and  clients,  and   (6)  on  the 
item  "efficient-inefficient,"  a  correlation  of  .39  be- 
tween the  ratings  of  student  counselors  and  clients. 
Although  these  correlations  between  groups  for  particular 
item  ratings  were  found  to  be  statistically  significant, 
it  is  important  to  be  aware  of  the  possibility  of  alpha 
errors.     One  hundred  and  fifty  correlations  of  item 
ratings  are  presented  in  Table  4 .     The  chances  of  a  false 
significant  difference  being  obtained  when  the  alpha 
level  is  set  at  .01  is  one  in  100,  while  at  .05  it  is 
five  in  100.     It  is  conceivable,  then,  that  the  signifi- 
cant correlations  for  particular  items  reported  above 
may  have  been  the  result  of  alpha  errors. 

In  Table  5,  the  means  and  standard  deviations  are 
presented  for  the  ratings  made  by  each  rating  group  on 
each  item  of  the  Counselor  Effectiveness  Scale.    A  one- 
way analysis  of  variance  was  performed  to  determine 
whether  the  mean  total  ratings  for  each  rating  group 
differed  significantly  from  those  of  the  other  rating 
groups.     As  may  be  seen  in  Table  6,  the  one-way  analysis 
of  variance  resulted  in  an  F  Score  of  2.331,  reflecting 
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TABLE  6 

Analysis  of  Variance  Comparing  Mean  Ratings 
of  the  Four  Rating  Groups 


Source  of  Variation 

SS 

df 

MS  F 

Between  Groups 
Within  Groups 

1533.7 
22806.0 

3 

104 

511.2     2.331  n.s. 
219.3 

Total  24339.7  107 
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a  nonsignificant  difference  among  the  mean  total  CES 
scores  for  the  four  rating  groups. 

Summary  of  Results 

In  summary,  none  of  the  six  null  hypotheses  was 
rejected.     The  data  indicate  that  there  were  no  signifi- 
cant relationships  among  overall  counselor  effectiveness 
ratings  made  by  student  counselors,  peers,  supervisors, 
and  clients  on  the  Counselor  Effectiveness  Scale,  Form 
2.     In  addition,  no  difference  was  found  among  the 
groups'  mean  ratings  on  the  total  score  of  the  Counselor 
Effectiveness  Scale.     In  other  words,  no  significant  re- 
lationships were  found  between  pairs  of  rating  groups  in 
terms  of  consistent  rating  patterns,  nor  were  the  overall 
student  counselor  ratings  made  by  each  of  the  four  rating 
groups  significantly  different. 

Nine  statistically  significant  correlations  (one 
at  p<.01,  eight  at  p<.05)  were  found  between  pairs  of 
rating  groups  for  ratings  on  particular  items.  However, 
because  of  the  high  possibility  of  alpha  error,  the 
validity  of  the  significant  correlations  for  specific 
items  is  questionable. 


CHAPTER  V 

SUMMARY,   DISCUSSION,  AND  FUTURE  RESEARCH 


Suimnary 

Purpose  of  the  Study 

Self,  peer,  supervisor,  and  client  ratings  have  all 
been  used  extensively  in  counseling  research  as  criteria 
of  effective  counseling.     However,  information  is  in- 
complete and  often  contradictory  regarding  the  extent 
to  which  these  different  rating  groups  agree  on  ratings 
of  counselor  effectiveness.     It  was  the  primary  purpose 
of  the  present  study  to  examine  the  consistency  of  coun- 
selor effectiveness  self-ratings,  peer  ratings,  super- 
visor ratings,  and  client  ratings  on  a  counselor  effec- 
tiveness scale.     The  secondary  purpose  of  the  study  was 
to  examine  the  consistency  of  ratings  made  by  these  groups 
of  raters  on  the  specific  items  within  the  scale. 

Instrumentation 

The  Counselor  Effectiveness  Scale,  Form  2,  was  the 
counselor  effectiveness  rating  instrument  used  in  the 
present  study.     It  is  a  semantic  differential  instrument 
containing  25  items,  with  each  item  having  seven  steps 
between  a  positive  counselor  trait   (e.g.,  helpful) 
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and  a  negative  counselor  trait  (e.g.,  unhelpful).  The 
CES  provides  a  global  measure  of  counselor  effectiveness 
and  is  appropriate  for  immediate  use  with  raters  of 
varying  sophistocation. 

Design 

Supervisors  in  the  Department  of  Counselor  Education 
at  the  University  of  Florida  and  in  various  counselor 
education  practicum  and  internship  agencies  rated  a 
randomly  selected  counselor  education  graduate  student 
supervisee  on  the  Counselor  Effectiveness  Scale,  Form  2. 
These  student  counselor  supervisees  were  also  rated  by 
themselves,  randomly  selected  peer  student  counselors, 
and  by  every  client  counseled  by  a  supervisee  for  a  pre- 
determined week.     The  relationship  between  each  pair  of 
rating  group's  overall  CES  ratings  was  ascertained  by 
use  of  Pearson  product-moment  correlations.     The  agree- 
ment between  each  pair  of  group's  ratings  on  the  specific 
items  of  the  CES  was  also  determined  by  means  of  Pearson 
product-moment  correlations.     A  one-way  analysis  of 
variance  was  employed  to  test  whether  the  mean  ratings 
for  each  rating  group  differed  significantly  from  those 
of  each  of  the  other  rating  groups. 

Results 


The  major  finding  in  the  present  study  was  that  there 
were  no  significant  relationships  among  the  overall 
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counselor  effectiveness  ratings  made  by  counselors  them- 
selves, peers,  supervisors,  and  clients.     Of  150  correla- 
tions for  25  item  scores,  nine  were  found  to  be  statis- 
tically significant  (one  p<.01,  eight  p<.05).  However, 
because  of  the  great  likelihood  of  alpha  error,  no  attempt 
was  made  to  draw  inferences  from  any  correlations  for 
specific  items.     The  overall  mean  ratings  for  the  four 
groups  were  not  found  to  be  statistically  different. 

Discussion 

Conclusions  and  Implications 

The  major  finding  in  the  present  study  was  that 
counselors,  peers,  supervisors,  and  clients  did  not 
agree  on  overall  ratings  of  counselor  effectiveness  as 
measured  by  the  Counselor  Effectiveness  Scale,  Form  2. 
This  finding  seems  to  support  the  view  that  effectiveness 
may  indeed  be  in  the  "eye  of  the  beholder."     It  may  be, 
then,  that  different  groups  have  differing  definitions  of 
"effective  counseling"  growing  out  of  individual  orienta- 
tions, biases,  internal  anchors,  or  other  differences. 
On  the  other  hand,  it  simply  may  be  that  a  counselor's 
effectiveness  varies  greatly  from  time  to  time  and  from 
client  to  client,  making  it  difficult  for  different 
groups  to  agree  on  that  counselor's  "general"  effective- 
ness. 

Whatever  the  reasons,  the  results  of  this  study  con- 
tradict the  general  assumption  that  the  counselor 


55 


effectiveness  ratings  made  by  counselors,  peers,  super- 
visors, and  clients  are  comparable.     The  results  clearly 
indicate  that  there  were  no  significant  relationships 
among  the  ratings  made  by  these  groups.     The  current 
practice  of  comparing  counselor  effectiveness  studies 
which  use  different  rating  groups  does  not  appear  justi- 
fied.    Since  ratings  made  by  different  groups  may  not  be 
consistent  with  one  another,  studies  using  one  particular 
rating  group  may  only  be  comparable  to  other  studies 
using  the  same  category  of  raters. 

It  also  appears  unwise  to  assume  that  a  counselor 
effectiveness  study  employing  one  group  of  raters  will 
necessarily  result  in  the  same  findings  if  a  different 
rating  group  is  substituted.     If  the  results  of  studies 
vary  as  a  function  of  the  rating  group  employed,  it  is 
critical  for  researchers  to  take  into  account  the  group 
doing  the  rating.     If  this  consideration  is  neglected, 
researchers  may  inadvertently  bias  their  results  by  em- 
ploying one  group  of  raters  instead  of  another. 

These  results  have  implications  for  counselor 
training.     If  three  groups  of  counseling-oriented  indi- 
viduals (supervisors,  student  counselors,  and  peers) 
cannot  agree  on  who  are  effective  counselors,  what  direc- 
tion should  counselor  training  take?     Is  it  even  possible 
to  train  people  to  be  effective  counselors  when  "effec- 
tive counseling"  may  mean  many  things  to  many  people? 
The  results  of  this  study  amplify  the  need  to  examine 
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whether  effective  counseling  is  relative  or  absolute.  If 
members  of  varying  groups  can  be  consistent  in  identify- 
ing "effective  counseling,"  then  counselor  educators  will 
need  to  be  more  active  in  operationally  defining  the 
characteristics  of  effective  counseling.     If  effective 
counseling  is  found  to  be  purely  relative,  this  fact 
would  also  need  to  be  openly  acknowledged.     To  be  sure, 
further  discussion  and  research  is  needed  before  any 
definitive  conclusions  are  reached  regarding  the  absolute 
or  relative  nature  of  counselor  effectiveness. 

Comments 

Since  the  results  of  this  study  are  at  variance  with 
other  studies  finding  significant  correlations  between 
rating  groups,  it  seems  appropriate  to  examine  possible 
explanations.     The  rating  instrument  used  in  the  present 
study  may  be  one  reason.     For  example,  it  may  be  more 
difficult  to  obtain  intra-group  agreement  on  global 
effectiveness  ratings  than  on  ratings  of  specific  behav- 
iors or  taped  responses.     A  global  rating  scale  such  as 
the  Counselor  Effectiveness  Scale,  Form  2,  used  in  the 
present  study,  seems  likely  to  allow  more  room  for  the 
rater's  own  subjective  preferences  to  color  his  ratings 
than  would  a  scale  focusing  on  specific  behaviors.     It  is 
also  possible  that  insufficient  contact  was  established 
between  the  counselor  and  the  other  raters  to  expect  high 
agreement. 


57 


While  such  methodological  weaknesses  should  be  con- 
sidered in  discussing  the  results  of  the  present  study, 
one  strength  should  also  be  mentioned.     In  this  study, 
no  rater  rated  more  than  one  counselor  subject,  and  no 
two  individuals  from  the  same  rating  group  rated  the 
same  counselor  subject.     The  resulting  independency  of 
measurements  appears  to  strengthen  this  study.  However, 
independency  of  measurement  procedures  are  not  always 
found  in  studies  reporting  significant  correlations  among 
rating  groups.     It  is  not  known  to  what  extent  dependency 
of  measurements,  a  statistically  unsound  procedure,  might 
have  resulted  in  high  correlations  among  rating  groups 
in  other  studies. 

Idiographic  Observations 

The  present  researcher  personally  administered  the 
CES  to  the  student  counselors,  peers,  and  supervisors  in 
this  study.     Many  subjects  volunteered  their  impressions 
of  the  study,  and  of  the  CES  in  particular.     Since  the 
perceptions  of  these  subjects  offer  a  perspective  unattain 
able  elsewhere,  they  will  be  presented  here.  However, 
subjects'  views  should  be  considered  idiographic  in  nature 
as  they  may  or  may  not  reflect  the  views  of  the  total 
group . 

The  most  common  comment  came  from  peers  who  felt 
that  their  contact  with  the  student  counselor  subjects 
was  limited.     Several  peers  from  both  student  counselors' 
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group  supervision  sessions  and  work  settings  stated  that 
they  found  it  difficult  to  rate  the  student  counselors, 
since  they  had  not  heard  counseling  tapes  of  the  student 
counselors  nor  observed  them  in  a  counseling  role.  This 
lack  of  contact  with  the  student  counselor  in  a  counsel- 
ing role  also  seemed  to  add  to  the  difficulty  in  rating 
the  counselor  subject  as  a  "counselor"  and  not  as  a 
"person."    One  peer  stated  that  it  would  have  been  much 
easier  to  rate  a  tape  or  a  specific  segment  of  behavior 
than  to  rate  a  person  on  a  global  measure  such  as 
"counselor." 

A  rating  observation  reported  by  one  student  coun- 
selor subject  was  that  he  found  it  difficult  to  rate  him- 
self as  highly  as  he  believed  he  should  be  on  most  items. 
He  stated  that,  at  least  for  him,  there  was  an  unwritten 
"norm"  not  to  give  himself  a  maximum  rating  on  any  item. 
It  may  be  that  this  individual  tempered  his  own  ratings 
with  modesty  while  others  rating  him  may  have  felt  freer 
to  give  their  true  impressions. 

The  value  of  each  particular  score  may  also  vary  as 
a  function  of  the  rater's  internal  anchoring.     For  exam- 
ple, one  counselor's  self,  peer,  and  supervisor  total 
CES  ratings  were  all  approximately  140,  while  the  client 
rating  was  113.     The  client's  apparently  poor  perception 
of  her  counselor  was  not  substantiated  by  her  written 
remarks  at  the  bottom  of  the  form:     "Just  by  talking  to 
someone  who  gives  you  the  feeling  they  are  concerned  ... 
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can  make  all  the  difference.     This  is  the  feeling  he 
communicates  .   .   .   . "     This  client  may  have  seen  single 
item  ratings  of  "4"  and  "5"  as  favorably  as  another  indi- 
vidual's ratings  of  "6"  and  "7." 

Several  subjects  pointed  out  what  they  saw  as  defi- 
ciencies in  the  CES.     One  subject  believed  that  certain 
adjective  pairs,  such  as  "skillful-clumsy,"  "nice-awful," 
and  "erratic-stable,"  were  incompatible  since  the  pairs 
did  not  represent  corresponding  polarities.  Another 
subject  felt  that  some  adjective  pairs   (e.g.,  "nice- 
awful")  were  weighted  more  heavily  on  the  nagative  side. 
Also,  some  supposedly  negative  adjectives  were  said  to  be 
seen  as  positive.     For  example,  an  "inconsistent"  person 
was  seen  by  one  rater  as  flexible.     Another  person  saw 
"erratic"  as  being  positive  for  the  same  reason.  Simi- 
larly, the  positive  adjective  "nice"  was  seen  as  negative 
or  "sugar  coated"  by  one  rater. 

Future  Research 

In  the  present  study  no  relationships  were  found 
among  counselor  effectiveness  self -ratings ,  peer  ratings, 
supervisor  ratings  and  client  ratings.     Since  the  results 
of  this  study  are  not  consistent  with  previous  counselor 
effectiveness  research,  more  questions  than  answers  may 
have  been  generated.     Listed  below  are  recommendations 
for  future  research  logically  growing  out  of  the  results 
of  this  study: 


60 


1.  The  lack  of  significant  agreement  found  among 
rating  groups  should  not  be  taken  to  mean  that 
all  four  rating  groups  provided  invalid  ratings 
of  counselor  effectiveness.     Research  still 
needs  to  be  conducted  to  determine  the  extent 
to  which  each  rating  group's  counselor  effec- 
tiveness ratings  are  related  to  client  outcome 
measures . 

2.  The  present  study  used  a  global  rating  of  coun- 
selor effectiveness.  Research  needs  to  be  done 
with  a  variety  of  rating  scales  to  determine 

to  what  extent  agreement  among  groups  varies 
as  a  function  of  the  rating  scale  used.  Par- 
ticular attention  should  be  given  to  examining 
whether  ratings  of  more  specific,  behavioral 
aspects  of  counseling  (e.g.,  ratings  of  taped 
segments  of  interviews)  result  in  greater  agree- 
ment among  rating  groups  than  was  found  in  this 
study. 

3.  The  minimal  levels  of  contact  established  be- 
tween the  counselor  and  members  of  the  other 
rating  groups  may  not  have  been  sufficient  to 
result  in  consistency  among  group  ratings.  Re- 
search should  be  conducted  to  determine  if 
greater  contact  between  rating  groups  will  re- 
sult in  significant  agreement  in  ratings. 
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4.  Since  it  appears  that  different  groups  may  have 
their  own  views  of  what  constitutes  counselor 
effectiveness,  an  important  next  step  would  be 
to  study  these  differences.     What  are  the  unique 
views  of  counselor  effectiveness,  and  what 
biases  or  values  go  into  the  formulation  of 
these  views  for  specific  rating  groups? 

5.  Since  ratings  made  by  one  group  may  not  always 
be  consistent  with  the  ratings  made  by  other 
groups,  future  counselor  effectiveness  research 
should  consider  using  multiple  rating  groups. 
In  this  way,  rating  consistency  among  groups 
can  be  monitored.     At  the  same  time,  counselor 
effectiveness  can  be  determined  from  the  per- 
spective of  the  members  of  each  specific  rating 
group. 

6.  More  use  should  be  made  of  observable  client 
outcomes   (i.e.,  school  attendance,  academic 
grades,  acts  of  misconduct)  as  measures  of 
counselor  effectiveness.     Such  behavioral  mea- 
sures would  allow  the  researcher  to  differentiate 
"effective"  from  "ineffective"  counselors  with- 
out the  use  of  questionable  rating  procedures. 
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Your  name : 


Your  sex : 


Your  age: 


Counselor  being  rated: 


CES,  Form  2 


Directions;     This  is  a  study  of  the  counselor  ratings  made 
by  various  groups.     Please  rate  the  counselor  named 
above  by  making  a  check   (✓)  on  one  of  the  seven  spaces 
between  each  adjective  pair  on  the  scale  on  this  and  the 
following  page.     For  example,  if  you  see  this  person  as 
a  more  sensitive  than  insensitive  counselor ,  you  might 
rate  him  as  follows: 

sensitive  ^  insensitive 

Again,  please  make  a  response  to  each  adjective  pair, 
even  though  some  adjectives  may  not  seem  to  be  appro- 
priate. 


IMPORTANT :     Your  ratings  will  be  kept  strictly  confidential. 
In  no  way  will  these  ratings  be  used  in  the  evaluation 
of  the  counselor.     The  responses  will  be  used  solely  for 
the  purposes  of  this  research  project  and  will  be  pre- 
sented only  in  terms  of  group  data. 

To  the  client;     Your  ratings  will  not  be  seen  by  your 

counselor  and  will  in  no  way  be  used  in  the  counseling 

you  receive.     Likewise,  your  participation  or  nonpartici- 

pation  will  in  no  way  affect  the  counseling  you  receive. 

Thank  you  for  your  cooperation. 


skillful   :  ;  :  :  :  ;   clumsy 

competent   ;  :  :  :  :  ;   incompetent 

confusing   :       ;       :       :       ;       :  clear 


meaningful   ;  :  :  ;  ;  .         not  meaningful 


deep 
sympathetic 
close 
socially  inept 
decisive 
friendly 
realistic 
irritable 
passive 
insecure 
strong 
nice 
erratic 
consistent 
indifferent 
lazy 
mature 
inattentive 
social 
efficient 
helpful 
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shallow 

unsympathetic 

distant 

socially  adept 

indecisive 

hostile 

unrealistic 

pleasant 

active 

secure 

weak 

awful 

stable 

inconsistent 

conscientious 

industrious 

immature 

attentive 

anti-social 

inefficient 

unhelpful 
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COUNSELOR  DEMOGRAPHIC  DATA  SHEET 

Your  Name:   

Number  of  practicums  or  internships  previously  completed: 

practicums:   

internships:   

Number  of  years  of  previous  counseling  experience  (not 
practicums  or  internships) :   

Individual  Supervisor:   

Present  practicum  or  internship  agency:   


\ 
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